In military and civil command center applications, it is often desirable to share multimedia data from numerous realtime and recorded sources among users having client systems (also referred to as clients) distributed over a network. In addition, these applications can benefit from a capability for spatial planning collaboration using a common network and service infrastructure, for example the public Internet, as the network fabric. These applications can also benefit from a system that efficiently maps multimedia data sources over the available capabilities of the client systems. Examples of these applications include emergency management and military command and control.
There are several challenges to implementing systems to support these command and control applications, including adaptation of the multimedia data to the capabilities of the various distributed platforms, management of information about the multimedia data, and distribution of spatial models for use in spatial planning collaboration. One of the challenges of Internet communication is transferring data amongst platforms of differing capabilities. The client system is generally the system receiving the multimedia data. The bandwidth, processing and display capabilities of the client system platform may not be sufficient to handle the data being sent by the source of multimedia data.
One conventional solution to this problem is the conversion of multimedia data including still images, audio, and video into different communications and display formats that require less bandwidth for delivery to the different platforms. In the case in which the same type of data can be handled by the receiving system, but not at the same bandwidth, resolution, color depth, etc., this conversion process is known as “transcoding.” In the case in which the type of data that can be handled by the receiving system is different from the original data type, this conversion process is known as “modality transformation.”
A modality transformation is the conversion of communication modalities (such as speech, text, and full motion video) into a different modality, for example speech-to-text. In most cases, modality transformations convert data of one modality type into data of a second type that requires less bandwidth or computational resources for transmission and display. The goal of the transformation is to provide client systems having limited resources access to the same information available to clients having greater resources. Often, there are several modality transformation services available to perform a modality transformation, and the client selects the service to use.
A distributed multimodal collaboration system should handle users joining and departing the collaboration session,at arbitrary times, data sources becoming available and unavailable at arbitrary times, and notification of new users joining the collaboration of original data sources and transformation data services available at that time. The readers (clients) of the multimedia data need to know when the data being delivered is no longer valid, for example, when a live video program has ended. The clients need to know when to shut down displays and to close data sources. The distributed multimodal collaboration system that provides a virtual environment for collaboration (also referred to as a collaborative virtual environment or CVE) should have the capability to distribute consistent models, including a model of the background scene (for example, terrain), models for objects to be manipulated in that scene, and transformations describing the location, orientation, and behavior of these objects and of the scene viewpoint. Generally, a session begins when a user begins to use the CVE, and other users join forming the collaboration. In some situations, a means to arbitrate simultaneous updates to the models from a hierarchy of users having different roles in the collaboration is required. CVE collaboration is described in, García et al.; “MOVE: Component Groupware Foundations for Collaborative Virtual Environments;” Proceedings of the 4th international conference on Collaborative virtual environments; 2002, pages 55-62.
It would, therefore, be desirable to provide a distributed multimodal collaboration system having a system of “descriptors” that can be passed to client systems to inform the clients of the availability of multimedia data and collaborative sessions, and a protocol for the distribution and invalidation of these descriptors via a reliable and flexible distributed communication system.
It would also be desirable to provide a protocol for the distribution of scene graph edits. It would also be further desirable to provide a marketplace-based protocol which provides the infrastructure whereby various distributed services assess their capability to perform a modality transformation and then bid for a contract to perform the transformation.